{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "74684add",
   "metadata": {},
   "source": [
    "# Categorical Similarity Space\n",
    "\n",
    "CategoricalSimilaritySpace provides an elegant solution for modeling categorical data in vector search applications. It creates n-hot encodings — binary vectors where each dimension represents a category and multiple dimensions can be set to 1 simultaneously — making it perfect for items that belong to multiple categories (like a product classified as both \"Clothing\" and \"Sports\").\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to effectively use CategoricalSimilaritySpace to model relationships between categorized items. You'll learn:\n",
    "\n",
    "1. **Basic implementation**: Creating and querying a categorical similarity space\n",
    "2. **Handling unknown categories**: Managing categories outside your predefined taxonomy\n",
    "3. **Category weighting**: Implementing preference and exclusion patterns in searches\n",
    "4. **Negative filtering**: Creating clear distinctions between matches and non-matches\n",
    "\n",
    "These techniques are valuable for applications like product categorization, content tagging systems, and any scenario where you need precise control over how categorical relationships influence search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2df7c789",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install superlinked==37.5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0f476a4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from superlinked import framework as sl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84750ca4",
   "metadata": {},
   "source": [
    "Here is our sample data.\n",
    "In this tutorial, we will create a Superlinked space only for the \"category\" field,\n",
    "while keeping the \"name\" field as a convenient label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "24433ae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_initial = [\n",
    "    {\"id\": \"product-1\", \"name\": \"T-shirt\", \"category\": [\"Clothing\"]},\n",
    "    {\"id\": \"product-2\", \"name\": \"Running Shoes\", \"category\": [\"Clothing\", \"Sports\"]},\n",
    "    {\"id\": \"product-3\", \"name\": \"Yoga Mat\", \"category\": [\"Sports\"]},\n",
    "    {\"id\": \"product-4\", \"name\": \"Pijama\", \"category\": [\"Clothing\", \"Home\"]},\n",
    "    {\"id\": \"product-5\", \"name\": \"Pillows\", \"category\": [\"Home\"]},\n",
    "]\n",
    "\n",
    "categories = [\"Clothing\", \"Sports\", \"Home\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "62b6cbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Product(sl.Schema):\n",
    "    id: sl.IdField\n",
    "    name: sl.String\n",
    "    category: sl.StringList\n",
    "\n",
    "\n",
    "product = Product()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d28ed30",
   "metadata": {},
   "source": [
    "### Basic usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "36d92142",
   "metadata": {},
   "outputs": [],
   "source": [
    "category_space = sl.CategoricalSimilaritySpace(category_input=product.category, categories=categories)\n",
    "index = sl.Index(category_space)\n",
    "source: sl.InMemorySource = sl.InMemorySource(product)\n",
    "executor = sl.InMemoryExecutor(sources=[source], indices=[index])\n",
    "app = executor.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63cfc849",
   "metadata": {},
   "source": [
    "Now we'll embed the data and ingest the vectors into the VectorDatabase.\n",
    "For this example, we'll use InMemoryVectorDatabase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50c7ec74",
   "metadata": {},
   "outputs": [],
   "source": [
    "source.put(data_initial)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6dfcda79",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = (\n",
    "    sl.Query(index)\n",
    "    .find(product)\n",
    "    .similar(category_space.category, sl.Param(\"query_categories\"))\n",
    "    # or shorter\n",
    "    # .similar(category_space, sl.Param(\"query_categories\"))\n",
    "    .select_all()\n",
    "    # select_all returns all schema fields\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6f97b750",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name            category         id  similarity_score\n",
       "0        T-shirt          [Clothing]  product-1               1.0\n",
       "1  Running Shoes  [Clothing, Sports]  product-2               1.0\n",
       "2         Pijama    [Clothing, Home]  product-4               1.0\n",
       "3       Yoga Mat            [Sports]  product-3               0.0\n",
       "4        Pillows              [Home]  product-5               0.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Clothing\"])\n",
    "# gives the same result as\n",
    "# result = app.query(query, query_categories=\"Clothing\")\n",
    "\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2015591e",
   "metadata": {},
   "source": [
    "A similarity score of 1.0 indicates a perfect match\n",
    "where all query categories (in this case \"Clothing\") are present in the product.\n",
    "Products without the \"Clothing\" category receive a score of 0.0.\n",
    "This demonstrates how CategoricalSimilaritySpace creates a clear binary distinction\n",
    "between items that match the query category and those that don't."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d41e5d99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name            category         id  similarity_score\n",
       "0         Pijama    [Clothing, Home]  product-4               1.0\n",
       "1        T-shirt          [Clothing]  product-1               0.5\n",
       "2  Running Shoes  [Clothing, Sports]  product-2               0.5\n",
       "3        Pillows              [Home]  product-5               0.5\n",
       "4       Yoga Mat            [Sports]  product-3               0.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Clothing\", \"Home\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eb45472",
   "metadata": {},
   "source": [
    "When querying with multiple categories [\"Clothing\", \"Home\"], products containing both categories (like \"Pijama\") receive a perfect score of 1.0. Products with only one of the categories receive a score of 0.5, representing a 50% match. Products with neither category receive a score of 0.0. \n",
    "\n",
    "The similarity calculation essentially measures the intersection of categories between the query and the product, normalized by the number of categories in the query."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9daf1340",
   "metadata": {},
   "source": [
    "### Adding new categories outside of predefined list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d119ea",
   "metadata": {},
   "source": [
    "Let's add more products with categories outside our predefined list\n",
    "and observe how our application handles them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1fd65151",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_kitchen = [\n",
    "    {\"id\": \"product-6\", \"name\": \"Blender\", \"category\": [\"Kitchen\"]},\n",
    "    {\"id\": \"product-7\", \"name\": \"Cooking Apron\", \"category\": [\"Clothing\", \"Kitchen\"]},\n",
    "]\n",
    "\n",
    "source.put(data_kitchen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3b50bc3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0         Pijama     [Clothing, Home]  product-4               1.0\n",
       "1        T-shirt           [Clothing]  product-1               0.5\n",
       "2  Running Shoes   [Clothing, Sports]  product-2               0.5\n",
       "3        Pillows               [Home]  product-5               0.5\n",
       "4  Cooking Apron  [Clothing, Kitchen]  product-7               0.5\n",
       "5       Yoga Mat             [Sports]  product-3               0.0\n",
       "6        Blender            [Kitchen]  product-6               0.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Clothing\", \"Home\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05e4d13e",
   "metadata": {},
   "source": [
    "You can see that \"Cooking Apron\" received a 0.5 score\n",
    "because it contains half of the categories specified in the query,\n",
    "specifically \"Clothing\".\n",
    "\n",
    "Now let's see how it works with \"Kitchen\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d1672593",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0        Blender            [Kitchen]  product-6               1.0\n",
       "1  Cooking Apron  [Clothing, Kitchen]  product-7               1.0\n",
       "2        T-shirt           [Clothing]  product-1               0.0\n",
       "3  Running Shoes   [Clothing, Sports]  product-2               0.0\n",
       "4       Yoga Mat             [Sports]  product-3               0.0\n",
       "5         Pijama     [Clothing, Home]  product-4               0.0\n",
       "6        Pillows               [Home]  product-5               0.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Kitchen\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "090c6cb2",
   "metadata": {},
   "source": [
    "Interestingly, although \"Kitchen\" is not in our predefined categories list, the application returned all Kitchen products with a perfect score.\n",
    "Why did we define a categories list if it's not being used as expected?\n",
    "\n",
    "The explanation will follow shortly, but first let's try another search with \"Electronics\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0268445c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0        Blender            [Kitchen]  product-6               1.0\n",
       "1  Cooking Apron  [Clothing, Kitchen]  product-7               1.0\n",
       "2        T-shirt           [Clothing]  product-1               0.0\n",
       "3  Running Shoes   [Clothing, Sports]  product-2               0.0\n",
       "4       Yoga Mat             [Sports]  product-3               0.0\n",
       "5         Pijama     [Clothing, Home]  product-4               0.0\n",
       "6        Pillows               [Home]  product-5               0.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Electronics\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afabc263",
   "metadata": {},
   "source": [
    "The results show that despite \"Kitchen\" not being in our predefined categories list, the application returned all products with the \"Kitchen\" category with a similarity score of 1.0.\n",
    "\n",
    "By default, CategoricalSimilaritySpace treats all categories not in the predefined list as a special \"other\" category. This means both \"Kitchen\" and \"Electronics\" are mapped to this same \"other\" category in the vector space, explaining why they produce identical search results. The parameter `uncategorized_as_category=True` (default) controls this behavior, enabling the system to handle unknown categories without requiring redefinition of the embedding space."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8b4058e",
   "metadata": {},
   "source": [
    "### Handling uncategorized values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b2fef77a",
   "metadata": {},
   "outputs": [],
   "source": [
    "category_space = sl.CategoricalSimilaritySpace(\n",
    "    category_input=product.category,\n",
    "    categories=categories,\n",
    "    uncategorized_as_category=False,\n",
    "    # uncategorized_as_category was True by default\n",
    "    # now we change it to False\n",
    ")\n",
    "index = sl.Index(category_space)\n",
    "\n",
    "source_without_uncategorized: sl.InMemorySource = sl.InMemorySource(product)\n",
    "executor = sl.InMemoryExecutor(sources=[source_without_uncategorized], indices=[index])\n",
    "app = executor.run()\n",
    "\n",
    "source_without_uncategorized.put(data_initial + data_kitchen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "156e00ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = sl.Query(index).find(product).similar(category_space.category, sl.Param(\"query_categories\")).select_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0aba56d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0        T-shirt           [Clothing]  product-1               0.0\n",
       "1  Running Shoes   [Clothing, Sports]  product-2               0.0\n",
       "2       Yoga Mat             [Sports]  product-3               0.0\n",
       "3         Pijama     [Clothing, Home]  product-4               0.0\n",
       "4        Pillows               [Home]  product-5               0.0\n",
       "5        Blender            [Kitchen]  product-6               0.0\n",
       "6  Cooking Apron  [Clothing, Kitchen]  product-7               0.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Kitchen\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6860c4f8",
   "metadata": {},
   "source": [
    "Now with `uncategorized_as_category=False`, the system ignores any categories not in our predefined list. This explains why searching for \"Kitchen\" returns no relevant products (all similarity scores are 0). This configuration is useful when you want strict adherence to your predefined taxonomy and don't want to allow new categories to emerge without explicit updates to your system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ec5b8eab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0        T-shirt           [Clothing]  product-1               0.0\n",
       "1  Running Shoes   [Clothing, Sports]  product-2               0.0\n",
       "2       Yoga Mat             [Sports]  product-3               0.0\n",
       "3         Pijama     [Clothing, Home]  product-4               0.0\n",
       "4        Pillows               [Home]  product-5               0.0\n",
       "5        Blender            [Kitchen]  product-6               0.0\n",
       "6  Cooking Apron  [Clothing, Kitchen]  product-7               0.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# same for \"Electronics\"\n",
    "\n",
    "result = app.query(query, query_categories=[\"Electronics\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e735b370",
   "metadata": {},
   "source": [
    "### Weighting categories"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39311977",
   "metadata": {},
   "source": [
    "Sometimes you want to weight categories differently.\n",
    "For example, you want to prefer some categories and exclude others.\n",
    "You can do that by chaining `.similar` calls with different weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4c5d3360",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = (\n",
    "    sl.Query(index)\n",
    "    .find(product)\n",
    "    .similar(category_space.category, sl.Param(\"preferred_categories\"), weight=1.0)\n",
    "    .similar(category_space.category, sl.Param(\"excluded_categories\"), weight=-1.0)\n",
    "    .select_all()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e30024df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>-0.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0  Running Shoes   [Clothing, Sports]  product-2               1.0\n",
       "1        T-shirt           [Clothing]  product-1               0.5\n",
       "2       Yoga Mat             [Sports]  product-3               0.5\n",
       "3  Cooking Apron  [Clothing, Kitchen]  product-7               0.5\n",
       "4         Pijama     [Clothing, Home]  product-4               0.0\n",
       "5        Blender            [Kitchen]  product-6               0.0\n",
       "6        Pillows               [Home]  product-5              -0.5"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, preferred_categories=[\"Clothing\", \"Sports\"], excluded_categories=[\"Home\"])\n",
    "\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f62acf99",
   "metadata": {},
   "source": [
    "We get the following interesting results:\n",
    "\n",
    "- \"Running Shoes\" received a 1.0 score because it's a perfect match with both preferred categories.\n",
    "\n",
    "- \"T-shirt\", \"Yoga Mat\", and \"Cooking Apron\" each received 0.5\n",
    "  because they contain only one of the two preferred categories\n",
    "  and none of the excluded categories.\n",
    "\n",
    "- \"Blender\" received 0.0 because it has \"Kitchen\", which is outside our predefined categories list.\n",
    "\n",
    "- \"Pijama\" received 0.0 because it has one preferred category (\"Clothing\")\n",
    "  but also has one excluded category (\"Home\"),\n",
    "  resulting in a score of 0.5 - 0.5 = 0.0\n",
    "\n",
    "- \"Pillows\" received -0.5 because it only contains the \"Home\" category, which we're excluding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0924cc71",
   "metadata": {},
   "source": [
    "Here is a more advanced scenario:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b9da51d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = (\n",
    "    sl.Query(index)\n",
    "    .find(product)\n",
    "    .similar(category_space.category, sl.Param(\"preferred_categories\"), weight=sl.Param(\"preferred_weight\"))\n",
    "    .similar(category_space.category, sl.Param(\"excluded_categories\"), weight=sl.Param(\"excluded_weight\"))\n",
    "    .select_all()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8a4c0aec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0  Running Shoes   [Clothing, Sports]  product-2               1.0\n",
       "1        T-shirt           [Clothing]  product-1               0.5\n",
       "2       Yoga Mat             [Sports]  product-3               0.5\n",
       "3  Cooking Apron  [Clothing, Kitchen]  product-7               0.5\n",
       "4        Blender            [Kitchen]  product-6               0.0\n",
       "5         Pijama     [Clothing, Home]  product-4              -1.5\n",
       "6        Pillows               [Home]  product-5              -2.0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(\n",
    "    query,\n",
    "    preferred_categories=[\"Clothing\", \"Sports\"],\n",
    "    preferred_weight=1.0,\n",
    "    excluded_categories=[\"Home\"],\n",
    "    excluded_weight=-4.0,\n",
    ")\n",
    "\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "876c04b4",
   "metadata": {},
   "source": [
    "By increasing the negative weight to -4.0 for excluded categories, we significantly penalize products in those categories. Note how \"Pillows\" now has a score of -2.0 and \"Pijama\" has -1.5, reflecting the stronger penalty for having the \"Home\" category."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9b264b8",
   "metadata": {},
   "source": [
    "### Negative filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758c1b25",
   "metadata": {},
   "source": [
    "If you want to generally downweight categories not mentioned in the query, you can set `negative_filter` to a negative number.\n",
    "This will push non-matching categories significantly lower in the results order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "17338886",
   "metadata": {},
   "outputs": [],
   "source": [
    "category_space = sl.CategoricalSimilaritySpace(\n",
    "    category_input=product.category,\n",
    "    categories=categories,\n",
    "    uncategorized_as_category=False,\n",
    "    negative_filter=-10,  # Strong penalty for non-matches\n",
    ")\n",
    "index = sl.Index(category_space)\n",
    "\n",
    "source_with_negative_filter: sl.InMemorySource = sl.InMemorySource(product)\n",
    "executor = sl.InMemoryExecutor(sources=[source_with_negative_filter], indices=[index])\n",
    "app = executor.run()\n",
    "\n",
    "source_with_negative_filter.put(data_initial + data_kitchen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "20d30492",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = sl.Query(index).find(product).similar(category_space.category, sl.Param(\"query_categories\")).select_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c620cfcf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>similarity_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pijama</td>\n",
       "      <td>[Clothing, Home]</td>\n",
       "      <td>product-4</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>T-shirt</td>\n",
       "      <td>[Clothing]</td>\n",
       "      <td>product-1</td>\n",
       "      <td>-8.160254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Running Shoes</td>\n",
       "      <td>[Clothing, Sports]</td>\n",
       "      <td>product-2</td>\n",
       "      <td>-8.160254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pillows</td>\n",
       "      <td>[Home]</td>\n",
       "      <td>product-5</td>\n",
       "      <td>-8.160254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cooking Apron</td>\n",
       "      <td>[Clothing, Kitchen]</td>\n",
       "      <td>product-7</td>\n",
       "      <td>-8.160254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Yoga Mat</td>\n",
       "      <td>[Sports]</td>\n",
       "      <td>product-3</td>\n",
       "      <td>-17.320508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Blender</td>\n",
       "      <td>[Kitchen]</td>\n",
       "      <td>product-6</td>\n",
       "      <td>-17.320508</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name             category         id  similarity_score\n",
       "0         Pijama     [Clothing, Home]  product-4          1.000000\n",
       "1        T-shirt           [Clothing]  product-1         -8.160254\n",
       "2  Running Shoes   [Clothing, Sports]  product-2         -8.160254\n",
       "3        Pillows               [Home]  product-5         -8.160254\n",
       "4  Cooking Apron  [Clothing, Kitchen]  product-7         -8.160254\n",
       "5       Yoga Mat             [Sports]  product-3        -17.320508\n",
       "6        Blender            [Kitchen]  product-6        -17.320508"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = app.query(query, query_categories=[\"Clothing\", \"Home\"])\n",
    "sl.PandasConverter.to_pandas(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13574d6f",
   "metadata": {},
   "source": [
    "The `negative_filter=-10` parameter dramatically changes how similarity scores are calculated. With this setting:\n",
    "\n",
    "1. Perfect matches (containing all query categories) still receive a score of 1.0\n",
    "2. Partial matches or mismatches receive substantial penalties\n",
    "3. The specific values (-8.16, -17.32) are derived from vector distance calculations with the negative filter applied\n",
    "\n",
    "This approach effectively creates a \"cliff\" in the similarity scores, making it easy to separate exact matches from everything else. It's particularly useful for applications where you need strict filtering but still want to maintain a ranking among the non-matching items."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3df0e3",
   "metadata": {},
   "source": [
    "### Summary: Key Features of CategoricalSimilaritySpace\n",
    "\n",
    "In this notebook, we've explored the CategoricalSimilaritySpace, a powerful tool for handling categorical data in vector search applications. Here's a summary of its key features:\n",
    "\n",
    "1. **N-hot encoding**: Transforms categorical data into binary vectors where each dimension represents a category, allowing items to belong to multiple categories simultaneously.\n",
    "\n",
    "2. **Flexible category handling**: \n",
    "   - With `uncategorized_as_category=True` (default), unknown categories are grouped into a special \"other\" category\n",
    "   - With `uncategorized_as_category=False`, unknown categories are ignored, enforcing strict adherence to predefined taxonomies\n",
    "\n",
    "3. **Powerful weighting options**:\n",
    "   - Apply positive weights to preferred categories\n",
    "   - Apply negative weights to excluded categories\n",
    "   - Use granular control with parameterized weights for fine-tuned relevance\n",
    "\n",
    "4. **Negative filtering**: Set `negative_filter` to a negative value to create a sharp distinction between exact matches and partial matches, effectively implementing \"must-have\" category requirements while maintaining ranking among non-matches.\n",
    "\n",
    "CategoricalSimilaritySpace is particularly useful for e-commerce product categorization, content tagging systems, and any application where items can belong to multiple discrete categories and where you need precise control over how these categories influence search results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
